On-Line Cursive Handwriting Recognition Using Hidden Markov Models and Statistical Grammars

The BYBLOS continuous speech recognition system is applied to on-line cursive handwriting recognition. By exploiting similarities between on-line cursive handwriting and continuous speech recognition, we can use the same base system adapted to handwriting feature vectors instead of speech. The use of hidden Markov models obviates the need for segmentation of the handwritten script sentences before recognition. To test our system, we collected handwritten sentences using text from the ARPA Airline Travel Information Service (ATIS) and the ARPA Wall Street Journal (WSJ) corpora. In an initial experiment on the ATIS data, a word error rate of 1.1% was achieved with a 3050-word lexicon, 52-character set, collected from one writer. In a subsequent writer-dependent test on the WSJ data, error rates ranging between 2%-5% were obtained with a 25,595word lexicon, 86-character set, collected from six different writers. Details of the recognition system, the data collection process, and analysis of the experiments are presented. I . I N T R O D U C T I O N The segmentation of written words into component characters is often the first step of handwriting recognition systems [1]. In some cases, segmentation is forced on the user by providing boxes for the writing of discrete letters. However, in modem continuous speech recognition efforts, segmentation of phonemes is not performed before either of the training or the recognition steps. Instead, segmentation occurs simultaneously with recognition. If such a system could be adapted for handwriting, the very difficult and time consuming issue of segmentation could be avoided. This paper addresses such a system, where automatic recognition of on-line cursive handwriting is achieved by the use of continuous speech recognition methods. In this context, on-line refers to the situation where the time sequence of samples comprising the script is known (as with pen computers). The recognition of the on-line handwriting is performed through the use of hidden Markov models and statistical grammars in a manner very similar to several modem speech recognizers. In fact, we show that, with essentially no modification, a speech recognition system can perform accurate on-line handwriting recognition with the input features being those of writing instead of speech. Hidden Markov models have intrinsic properties which make them very attractive for handwriting recognition. For training, all that is necessary is a data stream and its transcription (the text matching the handwriting). The training process automatically aligns the components of the transcription to the data. Thus, no special effort is needed to label training data. Segmentation, in the traditional sense, is avoided altogether. Recognition is performed on another data stream. Again, no explicit segmentation is necessary. The segmenteui~rently with the MIT Media Lab. tation of words into characters or even sentences into words occurs naturally by incorporating the use of a lexicon and a language model into the recognition process. The result is a text stream that can be compared to a reference text for error calculation. Section 2 discusses the similarities of speech and handwriting recognition tasks and provides some background on technique. Section 3 describes an initial 3050 word, 52 symbol, writer dependent experiment. Section 4 discusses a more ambitious 25,595 word, 86 symbol, writer dependent system involving multiple writers. Section 5 examines experimental results and discusses future work. 2 . C O M P A R I S O N O F C O N T I N U O U S S P E E C H R E C O G N I T I O N T O O N L I N E H A N D W R I T I N G R E C O G N I T I O N On-line handwriting and continuous speech share many common characteristics. On-line handwriting can be viewed as a signal (x,y coordinates) over time, just like in speech. The items to be recognized are well-defined (usually the alphanumeric characters) and finite in number, as are the phonemes in speech. The shape of a handwritten character depends on its neighbors. Correspondingly, spoken phonemes change due to coarticulation in speech. In both cases, these basic units form words and the words form phrases. Thus, language modeling can be applied to improve recognition performance for both problems. In spite of the above similarities, handwriting recognition has some basic differences to speech recognition. Unlike continuous speech, word boundaries are usually distinct in handwriting. Thus, words should be easier to distinguish. However, in cursive writing the dots and crosses involved in the characters "i", "j", "x", and "t" are not added until after the whole word is written. Thus, all the evidence for a character may not be contiguous. Additionally, in words with multiple crossings ("t" and "x") and/or dottings ( 'T' and "j") the order of pen strokes is ambiguous. Even so, with the many parallels between on-line writing and speech, speech recognition methods should be applicable to on-line handwriting recognition. Since hidden Markov models currently constitute the state of the art in speech recognition, this method also seems a likel3~ candidate for handwriting recognition. There has been some interest in the use of HMMs for on-line handwriting recognition (see, for example, [2, 3]). However, the few studies that have used HMMs have dealt with small vocabularies, isolated characters, or isolated words. In this study, our objective is to deal with continuous cursive handwriting and large vocabularies (thousands of words) using a speech recognition system and language models.